1 Field of the Invention
This invention relates generally to light field 3D imaging systems and methods thereof, such as image and video compression, decompression and modulation of light field image data used as input. The term “light field” describes the transmission and modulation of the light including, direction, amplitude, frequency and phase, therefore encapsulates imaging systems that utilize techniques such as holography, integral imaging, stereoscopy, multi-view imaging, Free-viewpoint TV (FTV) and the like.
2. Prior Art
In the present invention the holographic element or “hogel” is defined as the smallest unit of a sampled light field image and which contains information that can be directionally modulated by the 3D display to all available directions. Light field images can be represented as a 2D image matrix of hogels. The input images usually exhibit ample inherent correlation between hogels, which has been exploited in prior art (see M. Lucente, Diffraction-Specific Fringe Computation for Electro-Holography, Doctoral Thesis Dissertation, MIT Depart. of Electrical Engineering and Computer Science, September 1994, Ohm, J.-R., “Overview of 3D video coding standardization,” In International Conference on 3D Systems and Applications, Osaka, 2013, Heun-Yeung Shum et al., “Survey of image-based representations and compression techniques,” Circuits and Systems for Video Technology, IEEE Transactions on, vol. 13, no. 11, pp. 1020-1037, November 2003 and Kundu, S. “Light field compression using homography and 2D warping,” 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1349-1352, 25-30 Mar. 2012), with compression algorithms to reduce image data sizes.
To improve the compression of light fields, new 3D video coding standards are considering the adoption of techniques from the field of computer vision (ISO/IEC JTC1/SC29/WG11, Call for Proposals on 3D Video Coding Technology, Geneva, Switzerland, March 2011). With the use of per-pixel depth, reference images can be projected to new views, and the synthesized images can be used instead of the costly transmission of new images. This technique requires an increased amount of computational resources and local memory at the decoder side, posing a challenge for its real-time implementation. The 3D video compression tools are also targeting their use in horizontally arranged sequences, and do not exploit the 2D geometric arrangement of light fields. Methods developed exclusively for light field image compression include a vector quantization method described by Levoy et al (“Light Field Rendering,” Proceedings of the 23rd annual conference on Computer Graphics and Iteractive Techniques, SIGGRAPH 96), and video compression-based methods described by Magnor et al (Data Compression for Light-Field Rendering, IEEE Transaction on Circuits and Systems for Video Technology, v. 10, n. 3, April 2000, pp. 338-343). The use of vector quantization is limited and cannot achieve high compression performances such as those presented by Magnor et al. Their proposed methods are similar to a multiview compression algorithm, where the geometrical regularity of the images is exploited for disparity estimation. However, the proposed compression algorithms require an increased amount of local memory, and are not suited for real-time implementation. Furthermore, standard 3D video compression algorithms (Ohm, J.-R., “Overview of 3D video coding standardization,” In International Conference on 3D Systems and Applications, Osaka, 2013) or even specific light field compression methods (Heun-Yeung Shum et al., “Survey of image-based representations and compression techniques,” Circuits and Systems for Video Technology, IEEE Transactions on, vol. 13, no. 11, pp. 1020-1037, November 2003 and Kundu, S. “Light field compression using homography and 2D warping,” 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1349-1352, 25-30 Mar. 2012) cannot cope with the extremely large amounts of data generated by high-resolution light fields. As it can be appreciated by those skilled in the art, only a limited number of the compression methods described in the prior art can be implemented in real-time, and none of these methods can render and/or compress the amount of data required to drive a full parallax VAC-free display in real-time. Moreover, compression algorithms are usually designed for storage or network transmission (Bhaskaran, V. “65.1: invited Paper: Image/Video Compression—A Display Centric Viewpoint,” SID Symposium Digest of Technical Papers, vol. 39, no. 1, 2008), and in the case of a light field display system, the display has specific timing and memory requirements that cannot be fulfilled by conventional compression algorithms.
3D systems traditionally are limited by the capabilities of the display to handle the huge data requirements of light fields. Even when compression is employed, displays have to process the decoded data, the size of which can easily overwhelm display systems. Instead of applying compression, many light field display implementations resort to a reduction in the dimensionality of the light field at the source as a compromise to the increase in data. Nevertheless, limiting the resolution of light field displays can have a significant impact on the perceived quality and even cause visual fatigue. For example, super multiview displays, such as the ones presented in Takaki, Y., “High-density directional display for generating natural three-dimensional images,” Proc. IEEE, vol. 94, no. 3, pp. 654-663, March 2006, Balogh, T., “The HoloVizio system,” Proc. SPIE 6055, Stereoscopic Displays and Virtual reality Systems XIII, 60550U (Jan. 27, 2006), and Iwasawa, S. et al., “REI: an automultiscopic projection display,” Proc. 3DSA 2013, Selected paper 1, eliminate the vertical parallax of the light field, limiting the motion parallax to only horizontal movements. Integral Imaging displays (see Arai, J., “Three-dimensional television system based on integral photography,” Picture Coding Symposium (PCS), 2012, vol., no., pp. 17-20, 7-9 May 2012, Javidi, B., Seung-Hyun Hong, “Three-dimensional holographic image sensing and Integral Imaging display,” Display technology, Journal of, vol. 1, no. 2, pp. 341-346, December 2005, and Park, J. H., Hong, K. and Lee, B. “Recent progress in three-dimensional information processing based on integral imaging,” Applied optics 48, no. 34 (2009)) reproduce full parallax light fields, but are limited by the display resolution, and usually reduce the angular resolution (and consequently the depth-of-field) to increase the spatial resolution. Methods for holographic displays (see M. Lucente, Diffraction-Specific Fringe Computation for Electro-Holography, Doctoral Thesis Dissertation, MIT Depart. of Electrical Engineering and Computer Science, September 1994) resort to decreasing the display refresh rates in order to reduce transmission medium bandwidth. The works in Holliman, N. et al., “Three-Dimensional Displays: A Review and Application Analysis,” Broadcasting, IEEE Transactions on, vol. 57, no. 2, pp. 362-371, June 2011, Urey, H. et al., “State of the Art in Stereoscopic and Autostereoscopic Displays,” Proceedings of the IEEE, On page(s): 540-555 Volume: 99, Issue: 4, April 2011, and Masia, B. et al., “A survey on computational displays: Pushing the boundaries of optics, computation and perception,” Computers & Graphics 37.8 (2013) provide more examples of light field displays. However, those skilled in the art would immediately recognize that such techniques limit the capacity of a light field display to reproduce real 3D objects faithfully. The prior art fails to address the challenges imposed by high-resolution light field displays, such as high compression ratios, high quality, low computational load and real-time responses. Therefore, new methods and apparatus for high resolution light fields are required.